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Abstract 

We deal here with the issue of determinism versus randomness in time series. One 
wishes to identify their relative importance in a given time series. To this end we 
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extend i) the use of ordinal patterns-based probability distribution functions as- 
sociated to a time series [Bandt and Pompe, Phys. Rev. Lett. 88 (2002) 174102] 
and ii) the so-called Amigo paradigm of forbidden/missing patterns [Amigo, Zam- 
brano, Sanjuan, Europhys. Lett. 79 (2007) 50001], to analyze deterministic finite 
time series contaminated with strong additive noises of different correlation-degree. 
Insights pertaining to the deterministic component of the original time series are 
obtained with the help of the causal entropy-complexity plane [Rosso et al. Phys. 
Rev. Lett. 99 (2007) 154102]. 

PACS: 05.45.Tp; 02.50.-r; 05.40.-a; 05.40.Ca; 

Version: V14 



1 Introduction 

1.1 Preliminaries 

The distinction between deterministic and random components in time series 
has attracted considerable attention. From previous research, we may mention 
here work by A. R. Osborne and A. Provenzale [T|, G. Sugihara and R. May 
[2], D. T. Kaplan and L. Glass 01] • In particular, Kantz and co-workers 
[5|6] analyzed recently the behavior of entropy quantifiers as a function of 
coarse-graining resolution, and applied their ideas to the issue of trying to 
distinguish between chaos and noise. Why is this an important issue? Due 
to the concept of deterministic chaos, derived from the modern theory of 
nonlinear dynamical systems, has profoundly changed our thinking on time- 
series analysis. Emphasis is being placed on non-linear approaches, i.e., dealing 
with nonlinear deterministic autonomous equations of motion representative 
of chaotic systems that give birth to irregular signals [7]. 

Clearly, signals emerging from chaotic time series occupy a intermediate posi- 
tion between (a) predictable regular or quasi-periodic signals and, (h) totally 
irregular stochastic signals (noise) that are completely unpredictable. How- 
ever, they exhibit interesting phase-space structures. Chaotic systems display 
"sensitivity to initial conditions" which are the origin of instability everywhere 
in phase-space. This implies that instability uncovers information about the 
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phase-space "population", not available otherwise [8]. In turn this leads us to 
think of chaos as an information source, whose associated rate of generated- 
information is formulated in precise fashion via the Kolmogorov- Sinai's en- 
tropy piTU] . These considerations motivate our present interest in the compu- 
tation of quantifiers based on Information Theory, like, "entropy" , "statistical 
complexity", "entropy-complexity plane", etc. These quantifiers can be used 
to detect determinism in time series [11]. Indeed, different Information The- 
ory based measures (normalized Shannon entropy and statistical complexity) 
allow for a better distinction between deterministic chaotic and stochastic 
dynamics whenever "causal" information is incorporated via the Bandt and 
Pompe's (BP) methodology [12]. 



1.2 A new paradigm: forbidden patterns 



When nonlinear dynamics are involved, a deterministic system can generate 
"random-looking" results that nevertheless exhibit persistent trends, cycles 
(both periodic and non-periodic) and long-term correlations. Our main interest 
here, lies in the emergence of "forbidden/missing patterns" [T3|14fl5fl6fl7j . 
Why? Because they have the potential ability for distinguishing deterministic 
behavior (chaos) from randomness in finite time series contaminated with 
observational white noise [T^14fl7] . A concomitant helpful feature, as will be 
further explained below, is the decay rate of "missing ordinal patterns" as a 
function of the time series length. 

In fact, Zanin [TS] and Zunino et al. [IH] have recently studied the appearance 
of missing ordinal patterns in financial time series. The presence of missing 
ordinal patterns has also been recently construed as evidence of deterministic 
dynamics in epileptic states. Ouyang et al. [20] found that a missing pat- 
terns' quantifier could be used as a predictor of epileptic absence-seizures. It 
is essential to point out those works have only considered the presence of un- 
correlated noise (white noise), which makes the associated results somewhat 
incomplete, since the presence of colored noise might be of importance. A first 
step in such direction was recently trodden by us in pT]. We pursue matters 
here by investigating the robustness of an associated mathematical construct 
called the causal entropy-complexity plane [TT], that plays a prominent role 
in some of the above cited discoveries. We intend to do this by analyzing the 
planes's ability to distinguish between noiseless chaotic time series and the 
ones that are contaminated (weekly to very strongly) with additive correlated 
noise. The chaotic series studied here were generated by recourse to a logistic 
map to which noise with varying amplitudes was added. 
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1.3 Organization of the paper 

Our basic tools are 1) The MPR-statistical complexity measure together with 
entropic quantifiers, 2) The Bandt-Pompe approach to extract causal prob- 
ability distributions functions (PDFs) from a given time-series, and 3) the 
construction of an entropy-complexity plane. These subjects are covered in 
great detail in pT|21f22] . For completeness, some details are given in the Ap- 
pendix. The reader unfamiliar with these themes is strongly advised to go to 
the Appendix at this point. The forthcoming Section [2] details the problem 
to be discussed, while Section |3] revisits the Amigo paradigm. Our results are 
presented and discussed in Section HJ while some conclusions are drawn in 
Section O Background materials are provided in the Appendix. 



2 Logistic map plus observational noise 

The logistic map constitutes a canonic example, often employed to illustrate 
new concepts and/or methods for the analysis of dynamical systems. Here we 
will use the logistic map with additive correlated noise in order to exemplify 
the behavior of the normalized Shannon entropy l-Ls and the MPR-complexity 
Cj5, both evaluated using a PDF based on the Bandt-Pompe's procedure. We 
will also investigate the behavior of "missing ordinal patterns" in both a) 
the logistic map with additive noise (observational noise) and h) a. pure-noise 
series, where the number of time series data was fixed at A^. 

The logistic map is a polynomial mapping of degree 2, F : x„ — )■ x„+i [23], 
described by the ecologically motivated, dissipative system represented by the 
first-order difference equation 

3^n+l = ' Xn ■ {}- — Xn) , (1) 

with < < 1 and < r < 4. 

Let r]^''^ be a correlated noise with /"'^ power spectra generated as described 
in [TTI^ . The steps to be followed are enumerated below. 

(1) Using the Mersenne twister generator [21] through the Matlab® rand 
function we generate pseudo random numbers in the interval (—0.5, 0.5) 
with an (a) almost flat power spectra (PS), (b) uniform PDF, and (a) 
zero mean value. 

(2) The Fast Fourier Transform (FFT) y} is flrst obtained and then multi- 
plied by /~'^'^^, yielding y'f. Then, y^ is symmetrized so as to obtain a real 
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function. Subsequently the pertinent inverse FFT is found, after discard- 
ing the small imaginary components produced by the numerical approx- 
imations. The resulting noisy time series is then re-scaled to the interval 
[—1,1], which produces a new time series r]^^^ that exhibits the desired 
power spectra and, by construction, is representative of non-Gaussian 
noises. 

We consider time series of the form S = {Sn,n = 1, ■ ■ ■ , N} generated by the 
discrete system: 

Sn = Xr^ + A- ril'^ , (2) 

in which, x„ is given by the logistic map and r]^^ G [—1,1] represents a noise 
with power spectrum and amplitude A. 

In generating the logistic map's component of our time-series we fix r = 4 
and start the iteration procedure with a random initial condition. The first 
5 • 10^ iterations are considered part of the transient behavior and discarded. 
After the transient part dies out, = 10^ values are generated. In the case 
of the stochastic component of our time-series we consider < A; < 2 with 
fc-values changing by an amount Ak = 1 (other intermediate values have been 
considered in [21] )• The noise amplitude was varied in order to cover different 
regimes from weak (the noise can be consider as perturbation to the logistic) 
to very strong (the logistic is consider a perturbation to the noise). Ten noisy 
time-series of length = 10^ values and using different seeds, are generated 
for each value of the fc-exponent. 



3 The Amigo-paradigm 

We arrive at the central point of our discussion. For deterministic one dimen- 
sional maps, Amigo et al. [T3lll4fl5|[T6] have conclusively shown that not all 
the possible ordinal patterns (as defined using Bandt and Pompe's method- 
ology, see Appendix) can be effectively materialized into orbits, which in a 
sense makes these patterns "forbidden". We insist: this is an established fact, 
not a conjecture. The existence of these forbidden ordinal patterns becomes 
a persistent feature, a "new" dynamical property. For a fixed pattern-length 
(embedding dimension D) the number of forbidden patterns of a time series 
(unobserved patterns) is independent of the series length A^. It must be noted, 
that this independence does not characterize other properties of the series such 
as proximity and correlation, which die out with time [T1IT6]. For example, 
in the time series generated by the logistic map Xk+i = 4xfc(l — x^), if we 
consider patterns of length D = 3, the pattern {2, 1, 0} is forbidden. That is, 
the pattern Xk+2 < Xk+i < Xk never appears [Hj. 
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Stochastic processes could also have forbidden patterns [21]. However, in the 
case of uncorrelated (white noise) or certain correlated stochastic processes 
(noise with power low spectrum /"'^ with k > 0, ordinal Brownian motion, 
fractional Brownian motion, and fractional Gaussian noise), it can be numer- 
ically shown that no forbidden patterns emerge. In the case of time series 
generated by an unconstrained stochastic process (uncorrelated process) every 
ordinal pattern has the same probability of appearance [T3|14|15|IT6] . If the 
time series is long enough, all the ordinal patterns should eventually appear. 
If the number of time-series' observations is sufficiently big, the associated 
probability distribution function should be the uniform distribution, and the 
number of observed patterns should depend only on the length N of the time 
series under study. 

For correlated stochastic processes the probability of observing individual pat- 
terns depends not only on the time series length but also on the correlation 
structure [17] . The existence of a non-observed ordinal pattern does not qualify 
it as "forbidden" , only as "missing" , and is due to the finite length of the time 
series. A similar observation also holds for the case of real data series, as they 
always possess a stochastic component due to the omnipresence of dynamical 
noise [25|26|27j . The existence of "missing ordinal patterns" could be either 
related to stochastic processes (correlated or uncorrelated) or to deterministic 
noisy processes, which is the case for observational time series. 



3.1 The Carpi- Amigo test 

Amigo and co-workers [T3|T^ proposed a test that uses missing ordinal pat- 
terns to distinguish determinism (chaos) from pure randomness in finite time 
series contaminated with observational white noise (uncorrelated noise). The 
concomitant methodology [2] involves a graphic comparison between: 

• The decay rate of the missing ordinal patterns (of length D) of the time 
series under analysis as a function of the series length A^, and 

• the decay rate exhibited by white Gaussian noise. 

This methodology was recently extended by Carpi et al. [17] for the analysis 
of missing ordinal patterns in stochastic processes with different degrees of 
correlation. We are speaking of fractional Brownian motion (fBm), fractional 
Gaussian noise (fGn), and noises with power spectrum (PS) and k > 0. 
Results show that for a fixed pattern length, the decay rate of missing ordinal 
patterns in stochastic processes depends not only on the series length but also 
on their correlation structures. In other words, missing ordinal patterns are 
more persistent in the time series with higher correlation structures. Carpi et 
al. [IT] have also shown that the standard deviation of the estimated decay 
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rate of missing ordinal patterns (a) decreases with increasing D. This is due 
to the fact that longer patterns contain more temporal information and are 
therefore more effective in capturing the dynamic of time series with correla- 
tion structures. 

An important quantity for us, called A4{N,D), is the number of missing or- 
dinal patterns of length D not observed in a time series with N values. As we 
mentioned before, for pure correlated stochastic processes the probability of 
observing an individual pattern of length D depends on the time series-length 
and on the correlation structure (as determined by the type of noise k > 
). In fact, as established for noises with an f^^ PS in [lYj, as the value of 
k > augments - which implies that correlations grow - increasing values 
of A^ are needed in reaching the "ideal" condition Ai{N,D) = 0. If the time 
series is chaotic but has an additive stochastic component, then one expects 
that as the time series' length A^ increases, the number of "missing ordinal 
patterns" will decrease and eventually vanish. That this may happen does not 
only depend on the length A^ and on the underlying deterministic components 
of the time series but also on the correlation-structure of the added noise. 



4 Present results 



We wish to analyze and characterize the behavior of finite, noisy and chaotic 
time-series by recourse to patterns generated in the (causal) entropy-complexity 
plane. We intend to assess in particular the planar-geography of the forbid- 
den patterns. The focus of attention is centered upon the roles of i) the noise 
amplitude A and ii) the type of contaminating noise (degree of correlation). 
We consider noises with /"'^-power spectrum. 

For our present analysis, we fix the pattern-length at -D = 6, the embedding 
time lag at r = 1, and the time series length at A^ = 10^ values. For each one 
of the ten time series series generated (see Eq. (|2])) and for each pair {A, k), 
the normalized Shannon entropy Hs and the MPR-statistical complexity Cjs 
were evaluated using the Bandt and Pompe PDFs. We deal with additive 
(observational) noise with A; = 0, 1, and 2, and analyze their position (each 
point results from an average over 10 different series) in the Ti x C-planar 
graphs. A previous study has extensively dealt with the subject, but only in 
a special scenario in which the noise is of perturbative character [21]. Here 
we deal with the behavior of {A, k) in the planar-graphs way beyond the 
perturbative A-zone, by considering also A > 1 values. 
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4.1 CaseO<A<l 



Figure [T] summarizes the planar behavior of the noisy chaotic time series con- 
sidered (the logistic map with additive correlated noise with PS), con- 
sidering noise amplitudes < A < 1 {AA = 0.1) and k = (uncorrelated), 
k = 1,2 (correlated), see also Fig. 8 in ^21j. Clearly, given the noise- amplitude 
values here considered, the noise has a mere perturbative role vis-a-vis of the 
logistic map's chaotic behavior. 

For A = we obtain the purely deterministic value corresponding to the lo- 
gistic time-series, localized approximately at a medium-high entropic coordi- 
nate, near the highest possible complexity value. Note that such is the typical 
behavior observed for deterministic systems [llj. For a purely uncorrelated 
stochastic process {k = 0) we have Tis = 1 and Cjs = 0. The correlated (col- 
ored) stochastic processes {k ^ 0) yield points located at intermediate values 
between the curves Cmm and Cmax-, with decreasing values of entropy and in- 
creasing values of complexity as k grows yjj (see Fig. [H open symbols). From 
this figure and, also from our previous work |2T], we see that if 0, for 
increasing values of the amplitude A, entropy and complexity values change 
(starting from the value corresponding to the pure logistic series, i.e., A = 0) 
with a tendency to approach the values corresponding to pure noise, that is, 
("Hs ~ 1 and Cjs ~ 0). Similar behavior is observed when correlated noises 
are considered. A; 7^ 0. 

The main effect of the additive noise (A 7^ 0) is to shift the point representative 
of zero noise (A = 0) towards increasing values of entropy 1-is and decreasing 
values of complexity Cjs- This shift defines a kind of "trajectory" (curve) in 
the H, X C-plane, that is located in the vicinity of the maximum complexity 
Cmax— curve. Moreover, we found (by looking at other chaotic maps) that 
this trajectory is characteristic of the dynamical system under analysis. Such 
behavior can be linked to the persistence of forbidden patterns because they, 
in turn, imply that the deterministic logistic component is still operative and 
influencing the time series' behavior [21]. As k increases, the dependence on 
the noise- amplitude A < A < X) tends to become attenuated, implying 
that Ai{N,D) 7^ (see Fig. 4 of [21]), while it almost disappears for k ^ 2. 
This fact is due to the effect of the "coloring" correlations of the noise, that 
grow with increasing values of k. 



4.2 Case A > 1 

Here we are interested in the physics associated to A's growth, entailing an 
interchange between the roles of the logistic map and of the noise because a 
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large A makes noise the dominant feature. Several effects should be considered 

• noise contamination level (represented by A), 

• noise correlation (represented by k) and, 

• persistence of the forbidden patterns. 

We should expect that as A grows (with fixed values for N and D), the missing- 
patterns numbers would diminish and eventually vanish. With strong or very 
strong noise we expect zero missing patterns and eventual predominance of 
stochastic features. This means Ai{N,D) = independently of the A value. 
However, taking into account our previous results [21], we expect that some 
new specific features will be revealed by the use of the causality entropy- 
complexity plane. 

Consider our results for uncorrelated noise k = and increasing values of noise- 
amplitude A depicted in Fig. [2l This graph reveals that for A > 1 the points 
corresponding to increasing values of A continue to move in the plane towards 
the site representative of pure noise, with Ai{N,D) = 0. The perturbative 
character of the noise is, of course, lost. This behavior is apparent in Fig. 
[31 where the corresponding Bandt-Pompe PDF's, for a typical noisy chaotic 
time-series, are displayed for increasing noise amplitudes A. The corresponding 
value of A4{N, D) can also be observed in these graphs. 

At the top-left corner of Fig. [31 we depict the Bandt-Pompe's PDF for the 
unperturbed logistic map (A = 0). The presence of forbidden patterns (that 
have probabilities pi = 0) is clearly visible. In the same plot, at the bottom- 
right corner, we display the pure noise k = (white noise) PDF. No forbidden 
patterns exist now, and we observe the characteristic fiat distribution, pi = 
1/M for all patterns i = 1, ■ ■ ■ , M (M = D\ = 720). The noise- influence on the 
PDFs is clearly displayed in this graph (see also the plots for k = 1 and k = 2, 
Figures [3 and [7D . One appreciates the fact that the PDF evolves from that of 
the logistic map towards that of the pure- noise as one increases the value of A. 
The noise-effect also destroys the forbidden character of some patterns, whose 
associated probabilities grow slowly with the control parameter A, a feature 
that emphasizes the presence of a deterministic component in the noisy time 
series, even for J^{N, D) = 0, if A is not too large. This effect disappears for 
very strong noise. Note that for A = 5 the PDF is practically equal to that of 
pure noise. The two PDFs are almost indistinguishable aX A = 10, and their 
planar localizations coincide (see Fig. [2]). 

In Figures [H and [HI one can appreciate details of the planar-localization of 
noisy chaotic time-series (mean values taken over ten realizations) that emerge 
out of the noise-contaminated logistic map. We display values for correlated 
noise with k = 1 and k = 2, with increasing noise amplitude A > 1. In 
these plots we also observe the planar locations of both the time-series for the 
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unperturbed logistic map {A = 0) and the pure-correlated noise {k = 1,2). As 
in the previous case of uncorrelated noise {k = 0), the planar locations move 
towards the pure-noise's site (in the present cases, k = 1 and k = 2-pure noise, 
respectively) for increasing values of the noise-amplitude A. New behaviors 
emerge, as reflected by Figures H] and |6l For varying noise-amplitude values we 
appreciate two different trajectories: 1) displacement on the associated planar 
locations from left to right (starting form the unperturbed value A = 0), until 
reaching a point associated with a critical value A^. Ac = 7 for k = 1 (see Fig. 
H]) and Ac = 80 for /c = 2 (see Fig. E]). 2) At these critical values the trajectory 
reverses its direction (now it moves from right to left, "below" the first curve). 
The two trajectories converge at the planar location of the pure-correlated 
noise. Identical behavior was observed for all correlated noises with A;-values 
in the interval < < 2, that exhibit specific critical values of Ac{k). 

The planar behavior of the noisy, chaotic time-series can be interpreted in 
terms of two main interacting effects, namely, those associated with a) the per- 
sistence and robustness of the forbidden patterns in the deterministic chaotic 
component of the time-series and, b) the correlations characterizing stochas- 
tic observational noise (which in the present cases do not display forbidden 
patterns). These correlations grow with increasing values of A; > 0. Looking 
to the associated Bandt-Pompe's PDFs behavior, depicted in Figs. O and [7] 
for different values of A, the observed planar trajectories can be succinctly 
described as itemized below. 

• Amplitude range < A < A^. We deal with a two scenarios, one (Case A) 
in which the correlated noise acts as a perturbation {A4{N,D) ^ 0). In 
the other one (Case B) the logistic and the correlated noise have the same 
hierarchy {Ai{N,D) = 0). Fig. Ob shows that for k = 2 and case A the 
entropic and complexities' dispersion (given by the standard deviation) are 
both small and due to the perturbative noise. Contrariwise, for case B the 
dispersion values increase with the noise amplitude A. 

• Amplitude range A > A^. Now the correlated noise dominates. The deter- 
ministic component of the time series can be considered here as a perturba- 
tion. Again, for this range of noise amplitudes we have Ai{N,D) = 0. We 
also observe that the dispersions of the entropy Tig and the complexity Cjs 
are small. 



5 Conclusions 

We have revisited in some detail the paradigmatic concept of forbidden/missing 
ordinal patterns and used it as a tool for distinguishing between deterministic 
and stochastic behavior in empiric time-series. In the spirit of |13fl4fl5fl8fl9f20l 
we extended this kind of analysis by linking it to the physics described by the 
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causality entropy-complexity plane [TT]. Our considerations were made with 
regards to deterministic, finite time series contaminated with additive noises of 
different degree of correlation. Our analysis of the noise-determinism competi- 
tion clearly demonstrates that forbidden patterns are a deterministic feature of 
nonlinear systems and also reaffirms the usefulness of the entropy-complexity 
plane powerful tool of the theoretical arsenal. 

The planar localization in the causal entropy-complexity plane describing an 
uncontaminated, nonlinear deterministic system is displaced, by the addition 
of noise, towards a zone typical of pure stochasticity. This displacement gen- 
erates a kind of trajectory in the "H x C-plane that is located in the vicinity 
of the curve corresponding to maximum statistical complexity. 

If the noise is uncorrelated (white noise), the displacement of the systems 
characteristic point in the "H x C-plane becomes more pronounced as the noise 
intensity (amplitude) increases. It exhibits a monotonic decreasing behavior 
(from pure determinism to pure stochasticity). The noise-effect tends to elim- 
inate forbidden patterns. Since these, in turn, are features of a deterministic 
dynamics, their destruction signals deterministic-nature's loss. 

For contaminating correlated noise a new type of planar trajectory-behavior 
emerges as the noise intensity increases. Starting from a pure deterministic 
localization, the trajectory converges to a pure stochastic localization by fol- 
lowing a loop-curve as the noise intensity increases. For a critical value of 
noise intensity (Ac), however, this trajectory reverses direction. One observes 
a planar-movement of the systems characteristic point that starts at the un- 
perturbed value {A = 0) and closely approaches the curve of maximum com- 
plexity, from left to right, with increasing entropic values. At the critical noise 
intensity [Ac) this displacement reverses direction. It takes place now from 
right to left, below the original curve and converging to the planar location 
typical of pure-correlated noise. The value of the critical intensity Ac depends 
on the correlation degree of the noise and on the deterministic component. 

Three different scenarios can be associated: 

a) The correlated noise acting as a perturbation and Ai{N, D) ^ 0. The net 
noise effect is to destroy the forbidden character of some of the patterns. 
However due to the low noise intensity and to its correlations, the number 
of affected patterns is relatively low. The dominance of the deterministic 
component over the noisy one is reflected by low dispersion values for 
both the entropy and the statistical complexity. 

b) The deterministic and the stochastic components have the same hierar- 
chy and Ai{N, D) = 0. However, the persistent character of the forbidden 
patterns of the deterministic component and its interplay with the cor- 
relations present in the noise is reflected in the systems characteristic 
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point trajectory, that moves along the curve of maximum complexity, in- 
dicating that the pertinent patterns do not appear as frequently as the 
remaining ones. This behavior is indicative of a still active deterministic 
dynamics. The dispersion values for our two quantifiers increase with the 
noise intensity. 

The two scenarios above correspond to a noise intensity-range < A < Ac- 

c) The noisy component is the dominating one and the deterministic part 
can be considered as a perturbation. This scenario corresponds to the 
noise intensity range A > A^ and we have A4{N,D) = as well, with 
low dispersion values for entropy and statistical complexity. 



6 Appendix 

6.1 MPR- Statistical Complexity 

Given any arbitrary discrete probability distribution P = {pi : i = 1, - ■ ■ , M}, 
with M the number of freedom-degrees. Shannon's logarithmic information 
measure reads [28] 



The Shannon entropy is a measure of the uncertainty associated to the phys- 
ical process described by P. It is widely known that an entropic measure 
does not quantify the degree of structure or patterns present in a process [29j. 
Other measures of statistical or structural complexity, able capture their orga- 
nizational properties, are necessary for a better understanding of chaotic time 
series [30j. Our group introduced an effective statistical complexity measure 
(SCM) that is able to detect essential details of the dynamics and differentiate 
different degrees of periodicity and chaos [31]. This specific SCM, abbrevi- 
ated as the MPR-complexity one, provides important additional information 
regarding the peculiarities of the underlying probability distribution, not al- 
ready detected by the entropy. It is defined, following the seminal, intuitive 
notion advanced by Lopez- Ruiz et al. ^32j, via 




M 




(3) 



Cjs[P] = Qj[P,Pe]-'Hs[P], 



(4) 



usmg: 
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a) The normalized Shannon entropy (0 < Hs < 1) 

HsiP] = S[P]/Sma. , (5) 

where Smax = S[Pe] = InM and Pe = ■ ■ ■ , l/M} the uniform distri- 

bution. 

b) The so-called disequilibrium Qj (0 < Qj < 1), a quantifier defined in terms 
of the extensive (in the thermo dynamical sense) Jensen-Shannon divergence 
J[P, Pe] that links two PDFs. We have 

Qj[P,Pe]=Qo-J[P,Pe] , (6) 

with 

J[P, Pe] = S [(P + Pe)/2] - S[P]/2 - S[Pe]/2 . (7) 

Qo is a normalization constant, equal to the inverse of the maximum possible 
value of J'[P,Pe]. This value is obtained when one of the values of P, say 
Pm, is equal to one and the remaining pi values are equal to zero, i.e., 

Qo = -2|(^^^^^)ln(M + l)-21n(2M)+lnM| . (8) 

The Jensen-Shannon divergence, that quantifies the difference between two 
(or more) probability distributions, is especially useful to compare the symbol- 
composition of different sequences [33]. The complexity measure constructed 
in this way has the intensive property found in many thermodynamic quan- 
tities [31]. We stress the fact that the statistical complexity defined above is 
the product of two normalized entropies (the Shannon entropy and Jensen- 
Shannon divergence), but it is a nontrivial function of the entropy because it 
depends on two different probabilities distributions, i.e., the one corresponding 
to the state of the system, P, and the uniform distribution, Pg. 

6.2 Entropy- Complexity plane 

The time-evolution of the MPR-complexity can be analyzed using a diagram 
of Cjs versus time t. Thermodynamics' second law states that for isolated 
systems the entropy grows monotonically with time [dT-Ls/dt > 0), entailing 
that Hs can be viewed an "arrow of time" [51]. This suggests to study the 
temporal evolution of the SCM via an analysis of Cjs versus Tis- A normalized 
entropy-axis substitutes for the time-axis. Since one knows that for a given 
value of Tis the range of possible MPR-complexity values varies between a 
minimum Cmin and a maximum Cmax [35], our evolutionary path must take 
place in the planar-region delimited by these curves. It was demonstrated 
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that evolutionary path analysis generates insight into the details of the sys- 
tems probability distribution (not provided by randomness measures like the 
entropy [TPISOJ) that helps to uncover information related to the correlational 
structure between the components of a given physical process [SniEZ]- 



6.3 Estimation of the Probability Distribution Function 



Before attempting the time series characterization using Information Theory 
quantifiers, a probability distribution function (PDF) associated to the time 
series under analysis should be provided beforehand. The determination of 
the most adequate PDF is a fundamental problem because P and the sample 
space Q are inextricably linked. Many methods have been proposed for a 
proper selection of the probability space {Q, P). We can mention: (a) frequency 
counting |38], (b) procedures based on amplitude statistics |39], (c) binary 
symbolic dynamics |10], (d) Fourier analysis lU] and, (e) wavelet transform 
|42] . among others. Their applicability depends on particular characteristics of 
the data, such as stationarity, time series length, variation of the parameters, 
level of noise contamination, etc. In all these cases the dynamics' global aspects 
can be somehow captured, but the different approaches are not equivalent 
in their ability to discern all the relevant physical details. One must also 
acknowledge the fact that the above techniques are introduced in a rather "ad 
hoc fashion" and they are not directly derived from the dynamical properties 
themselves of the system under study. 



A better procedure is provided by the symbolic analysis of the time series, 
that discretizes the raw series and transforms it into a sequence of symbols. 
Symbolic analysis is efficient for nonlinear data processing, with low sensitivity 
to noise ^43j. Of course, meaningful symbolic representations of the original 
series are not provided by God and require some effort [HpS] . Today many 
people are confident that the Bandt and Pompe approach for generating PDF's 
is one of the most simple symbolization techniques and takes into account 
time-causality in the evaluation of the PDF associated to a time series p^. 
Symbolic data are i) created by associating a rank to the series- values and 
ii) defined by reordering the embedded data in ascending order. Data are 
reconstructed with an embedding dimension D. In this way it is possible to 
quantify the diversity of the ordering symbols (patterns) derived from a time 
series, evaluating the so called "permutation entropy" Us, and permutation 
statistical complexity Cjs (i.e., the normalized Shannon entropy and MPR- 
statistical complexity measure evaluated for the Bandt and Pompe's PDF). 
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6-4 The Bandt and Pompe approach for PDF construction 

Bandt and Pompe \X2l introduced a simple and robust method to evaluate 
the probability distribution taking into account the time causality of the sys- 
tem dynamics. They suggested that the symbol sequence should arise natu- 
rally from the time series, without any model-based assumptions. Thus, they 
took partitions by comparing the order of neighboring values rather than 
partitioning the amplitude into different levels. That is, given a time series 
S = {xt;t = 1,---,A^}, an embedding dimension D > 1 {D E N), and an 
embedding time delay r (r G N), the ordinal pattern of order D generated by 



is to be considered. To each time s we assign a D-dimensional vector that re- 
sults from the evaluation of the time series at times s — {D — l)r, ■ ■ ■ , s — r, s. 
Clearly, the higher the value of D, the more information about the past is 
incorporated into the ensuing vectors. By the ordinal pattern of order D 
related to the time s we mean the permutation vr = (ro, ri, ■ ■ ■ , r/)_i) of 
(0, 1, ■ ■ ■ , D - 1) defined by 

In this way the vector defined by Eq. is converted into a unique symbol it. 
In order to get a unique result we consider that < rj_i if x^-r^r = a;s_r,_iT- 
This is justified if the values of Xt have a continuous distribution so that equal 
values are very unusual. 

For all the D\ possible orderings (permutations) vTj when the embedding di- 
mension is D, their associated relative frequencies can be naturally computed 
by the number of times this particular order sequence is found in the time 
series divided by the total number of sequences. 



In the last expression the symbol jj stands for "number". Thus, an ordinal 
pattern probability distribution P = {p{7Ti),i = 1, ■ ■ ■ ,D\} is obtained from 
the time series. 

It is clear that this ordinal time-series' analysis entails losing some details of 
the original amplitude-information. Nevertheless, a meaningful reduction of 
the complex systems to their basic intrinsic structure is provided. Symboliz- 
ing time series, on the basis of a comparison of consecutive points allows for 




(9) 



< N — {D ~ l)r; (s) has type tTj} 
N-{D-1)t 



(11) 
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an accurate empirical reconstruction of the underlying phase-space of chaotic 
time-series affected by weak (observational and dynamical) noise [12] . Further- 
more, the ordinal-pattern probability distribution is invariant with respect to 
nonlinear monotonous transformations. Thus, nonlinear drifts or scalings ar- 
tificially introduced by a measurement device do not modify the quantifiers' 
estimations, a relevant property for the analysis of experimental data. These 
advantages make the BP approach more convenient than conventional meth- 
ods based on range partitioning. Additional advantages of the Bandt and 
Pompe method reside in its simplicity (we need few parameters: the pattern 
length/embedding dimension D and the embedding time lag r) and the ex- 
tremely fast nature of the pertinent calculation-process [i6P7] . We stress that 
the Bandt and Pompe's methodology is not restricted to time series repre- 
sentative of low dimensional dynamical systems but can be applied to any 
type of time series (regular, chaotic, noisy, or reality based), with a very weak 
stationary assumption (for k = D, the probabihty for Xt < Xt+k should not 
depend on t p^). 

The probability distribution P is obtained once we fix the embedding di- 
mension D and the embedding time delay r. The former parameter plays an 
important role for the evaluation of the appropriate probability distribution, 
since D determines the number of accessible states, given by D\. Moreover, it 
has been established that the length of the time series must satisfy the con- 
dition N ^ D\ in order to achieve a proper differentiation between stochastic 
and deterministic dynamics [llj. With respect to the selection of the param- 
eters, Bandt and Pompe suggest in their cornerstone paper [12] to work with 
3 < < 7 with a time lag r = 1. Nevertheless, other values of r might provide 
additional information. Soriano et al. [ISflQ] and Zunino et al. [50j recently 
showed that this parameter is strongly related, when it is relevant, with the 
intrinsic time scales of the system under analysis. In the present work D = Q 
and r = 1 are used. Of course it is also assumed that enough data are available 
for a correct embedding-time delay procedure (attractor-reconstruction). 
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Fig. 1. The causality entropy-complexity plane for all time series values (mean) 
corresponding to noise amplitude < vl < 1 (A^ = 0.1) and k = 0,1,2. Values 
for strictly noisy time series are also shown as open symbols. The values shown 
were obtained from times series of length N = 10^, and Bandt-Pompe parameters 
Z? = 6, r = 1. The lines represent the values of minimum and maximum statistical 
complexity Cmin and Cmax evaluated for the case of pattern length D = 6. 
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Fig. 2. a ) The causality entropy-complexity plane for all time series values (mean) 
corresponding to increasing values of noise amplitude A and /c = 0. Value for the case 
of strictly noisy time series is shown as open symbol. The values shown were obtained 
from times series of length N = 10^, and Bandt-Pompe parameters D = 6, r = 1. 
The lines represent the values of minimum and maximum statistical complexity 
Cmin and Cmax evaluated for the case of pattern length D = 6. h) Detail for high 
entropy values. 
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Fig. 3. The Bandt-Pompe PDF (M = 720) for a typical noisy chaotic time series 
considered (uncorrelated noise k = 0), for different increasing values of the noise 
amplitude A. In the top-left corner the PDF corresponding to the unperturbed 
logistic map {A = 0) is presented. In the bottom-right corner, the PDF for a pure 
noise k = time series is display. The values shown were obtained from times series 
of length N = 10^, and Bandt-Pompe parameters D = 6, t = 1. 
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Fig. 4. a) The causality entropy-complexity plane for all time series values (mean) 
corresponding to increasing values of noise amplitude A and k = \. Value for the case 
of strictly noisy time series is shown as open symbol. The values shown were obtained 
from times series of length = 10^, and Bandt-Pompe parameters D = 6, r = 1. 
The lines represent the values of minimum and maximum statistical complexity 
Cmin and Cmax evaluated for the case of pattern length Z) = 6. h) Detail for high 
entropy values. 
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Fig. 5. The Bandt-Pompe PDF (M = 720) for a typical noisy chaotic time series 
considered (correlated noise A; = 1), for different increasing increasing values of the 
noise amplitude A. In the top-left corner the PDF corresponding to the unperturbed 
logistic map (A = 0) is presented. In the bottom-right corner, the PDF for a pure 
noise k = 1 time series is display. The values shown were obtained from times series 
of length N = 10^, and Bandt-Pompe parameters D = 6, t = 1. 
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Fig. 6. a) The causality entropy-complexity plane for all time series values (mean) 
corresponding to increasing values of noise amplitude A and k = 2. Value for the case 
of strictly noisy time series is shown as open symbol. The values shown were obtained 
from times series of length N = 10^, and Bandt-Pompe parameters D = 6, r = 1. 
The lines represent the values of minimum and maximum statistical complexity 
Cmin and Cmax evaluated for the case of pattern length D = Q. h) Same, showing 
mean values and standard deviation for each A considered value. 
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Fig. 7. The Bandt-Pompe PDF (M = 720) for a typical noisy chaotic time series 
considered (correlated noise k = 2), for different increasing values of the noise 
amplitude A. In the top-left corner the PDF corresponding to the unperturbed 
logistic map {A = 0) is presented. In the bottom-right corner, the PDF for a pure 
noise k = 2 time series is display. The values shown were obtained from times series 
of length N = 10^, and Bandt-Pompe parameters D = Q, t = 1. 
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